Robert Downey Jr. had reservations about `Dolittle` even before it flopped

"We had this reset of priorities and made some changes in who our closest business advisers were," Downey Jr. added